Search engines, such as Google® and Bing®, are searchable indexes or directories of network sites. Although links may be manually submitted, a search engine typically has a crawler application that follows links embedded in network pages in order to find, retrieve, and index other network pages. Usually, the owner of a network site considers it desirable to have the pages of the network site indexed by the search engine crawler to drive traffic to the network site. However, mechanisms such as the robot exclusion standard and hypertext markup language (HTML) elements such as the “nofollow” attribute value for links or the “noindex” tag for pages are available to site owners for excluding pages from crawling and/or indexing.